{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "de765f16-341b-4dc3-b59d-b16437c7e050",
   "metadata": {},
   "source": [
    "\n",
    "## RAG\n",
    "\n",
    "Please refer to langchain guideline for [Retrieval](https://python.langchain.com/docs/modules/data_connection/)\n",
    "\n",
    "Vector store-backed retriever\n",
    "\n",
    "https://python.langchain.com/docs/modules/data_connection/retrievers/vectorstore\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e25cdbdc-daee-4a9b-b363-9080dc216927",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name: langchain\n",
      "Version: 0.1.9\n",
      "Summary: Building applications with LLMs through composability\n",
      "Home-page: https://github.com/langchain-ai/langchain\n",
      "Author: \n",
      "Author-email: \n",
      "License: MIT\n",
      "Location: /opt/conda/lib/python3.11/site-packages\n",
      "Requires: aiohttp, dataclasses-json, jsonpatch, langchain-community, langchain-core, langsmith, numpy, pydantic, PyYAML, requests, SQLAlchemy, tenacity\n",
      "Required-by: \n"
     ]
    }
   ],
   "source": [
    "!pip show langchain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e07b2c7d-62d7-457c-bbcb-298b29aa522c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Please define the following env variables in the .env file \n",
    "# \n",
    "# ollama_url = http://ollama:11434\n",
    "# model_name = mistral:instruct\n",
    "# embedding_model=nomic-embed-text\n",
    "# num_thread=8\n",
    "# "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fa2048ea-dbc4-49d7-b855-8b546dce2edb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the utility.py to get llm, embedding and output parser\n",
    "import utility\n",
    "embeddings = utility.get_embeddings()\n",
    "llm = utility.get_llm()\n",
    "output_parser = utility.get_output_parser()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1968c0c3-42cf-434c-800a-cfa084554e53",
   "metadata": {},
   "source": [
    "## Use existing LanceDB as the vector store"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b884ed11-1e88-4ccf-9111-787c0b606522",
   "metadata": {},
   "outputs": [],
   "source": [
    "# read from existing table\n",
    "import pyarrow as pa\n",
    "import lancedb\n",
    "from langchain_community.vectorstores import LanceDB\n",
    "\n",
    "db = lancedb.connect(\"lancedb\")\n",
    "table = db.open_table('structurizr')\n",
    "vectorstore = LanceDB(table, embeddings)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffccd7ce-cea2-495a-a565-7bc944e0d639",
   "metadata": {},
   "source": [
    "## Create the chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "02b3f72b-a7e1-48f9-8a08-e61399644775",
   "metadata": {},
   "outputs": [],
   "source": [
    "# retriever = vectorstore.as_retriever()\n",
    "retriever = vectorstore.as_retriever(search_kwargs={\"k\": 2})\n",
    "# retriever = vectorstore.as_retriever(search_type=\"similarity_score_threshold\", search_kwargs={\"score_threshold\": 0.5})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "081c8e88-e45c-4b32-bc0e-206801565f83",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.runnables import RunnableParallel, RunnablePassthrough\n",
    "\n",
    "retrieval = RunnableParallel(\n",
    "    {\"context\": retriever, \"input\": RunnablePassthrough()}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "dd527cf0-379d-4800-bff7-02b7cfdf41c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "\n",
    "prompt = ChatPromptTemplate.from_messages([\n",
    "    (\"system\", \"You are a software architect with knowledage in structurizr helping software architecture modeling and documenation in the following context: {context}\"),\n",
    "    (\"user\", \"{input}\")\n",
    "])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "fd5a6f70-6f06-4360-b35f-8a5509c36bc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = retrieval | prompt | llm | output_parser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ca5bda37-eb44-4f61-922c-573115676d19",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Does company XYZ use structurizr dsl\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a73baec-9caf-48fd-b460-33466dd5a9a5",
   "metadata": {},
   "source": [
    "## Similarity Search using vectorstore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7e8f145f-109b-4987-9ee0-6107b483c088",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "************************************\n",
      "Some famous companies such as ABC and XYZ are using Structurizr DSL for software architecture modeling and documentation.\n",
      "************************************\n",
      "Please see the DSL cookbook for a tutorial guide to the Structurizr DSL.\n",
      "************************************\n",
      "The Structurizr DSL provides a way to define a software architecture model\n",
      "(based upon the C4 model) as text, using a domain specific language (DSL).\n",
      "The DSL is rendering tool independent, and to render diagrams you will need to use one of the tools listed at\n",
      "Structurizr DSL - Rendering tools.\n",
      "************************************\n",
      "The named parameters are then available via the getParameter(name) method of the StructurizrDslPluginContextobject.\n"
     ]
    }
   ],
   "source": [
    "for doc in vectorstore.similarity_search(query):\n",
    "    print(f'************************************\\n{doc.page_content}', flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ca89820-ef4d-47e8-81e0-cc5de75134dc",
   "metadata": {},
   "source": [
    "## Similarity Search using retriever"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f57741b5-9473-4ec8-a141-fc664b9f4b55",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "************************************\n",
      "Some famous companies such as ABC and XYZ are using Structurizr DSL for software architecture modeling and documentation.\n",
      "************************************\n",
      "Please see the DSL cookbook for a tutorial guide to the Structurizr DSL.\n"
     ]
    }
   ],
   "source": [
    "for doc in retriever.get_relevant_documents(query):\n",
    "    print(f'************************************\\n{doc.page_content}', flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e48d485-6d25-4101-908f-9091954de75e",
   "metadata": {},
   "source": [
    "## Invoke LLM chain\n",
    "\n",
    "Notes: the chain can't answser the query correctly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "1f72f730-2bd2-43a3-800c-517d36b31458",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start at    ... 2024-03-12 01:30:38.013888\n",
      " I cannot directly answer that question as I don't have access to internal information about company XYZ. However, based on the provided code snippet, it seems that this code is using a library called 'structurizr' for modeling business processes or structures. The 'dsl' term in this context could refer to 'Domain Specific Language', and Structurizr does indeed provide a Domain Specific Language (DSL) for describing process models. So, based on the code alone, it appears that the company XYZ might be using structurizr DSL for modeling their business processes or structures. However, this is just an educated guess and further investigation would be required to confirm this.\n",
      "Complete at ... 2024-03-12 01:35:00.228868\n",
      "Total time used 262.21403884887695 seconds\n",
      "\n"
     ]
    }
   ],
   "source": [
    "time1 = utility.chain_starts()\n",
    "\n",
    "for chunk in chain.stream(query):\n",
    "        print(chunk, end=\"\", flush=True)\n",
    "\n",
    "utility.chain_completes(time1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6da3b66c-6217-4cd0-be5a-b589762b052c",
   "metadata": {},
   "source": [
    "\n",
    "## Change the Prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a9f922b8-2fb5-417f-8f2f-2475e48b482d",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = ChatPromptTemplate.from_messages([\n",
    "    (\"system\", \"\"\"You are a world class software architecture designer with knowledage of structurizr in the following context:\n",
    "    {context}\n",
    "\n",
    "    Please answer the question in the above context only, please do not make up anything if you do not know the answer.\n",
    "    \"\"\"),\n",
    "    (\"user\", \"{input}\")\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "42290b74-173f-40fe-a96d-99c70096c0e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = retrieval | prompt | llm | output_parser"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1aeee010-843c-4563-ba8a-781d4ce501bd",
   "metadata": {},
   "source": [
    "## Invoke LLM chain\n",
    "Notes: the chain can't answser the query correctly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "2f5062ce-9e82-4167-ab2e-040be367aa29",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start at    ... 2024-03-12 01:35:00.248960\n",
      " Based on the provided code snippet, it does not appear that Company XYZ uses Structurizr DSL directly. The code seems to be processing a graph represented as a list of nodes and edges, which is then likely used to generate or manipulate a diagram using Structurizr's library. However, without more context, it's impossible to confirm with certainty if Structurizr is being used for visualization purposes only or if it's part of the company's internal data processing pipeline.\n",
      "Complete at ... 2024-03-12 01:39:11.859737\n",
      "Total time used 251.61001110076904 seconds\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "time1 = utility.chain_starts()\n",
    "\n",
    "for chunk in chain.stream(query):\n",
    "        print(chunk, end=\"\", flush=True)\n",
    "\n",
    "utility.chain_completes(time1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd108c61-edda-41a4-a811-aa2939f0ebd9",
   "metadata": {},
   "source": [
    "## ConversationalRetrievalChain\n",
    "\n",
    "https://python.langchain.com/docs/integrations/providers/vectara/vectara_chat#conversationalretrievalchain-with-question-answering-with-sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "32e2f39c-6c4b-4ed4-8aa2-253d9967f6e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import LLMChain\n",
    "from langchain.chains import ConversationalRetrievalChain\n",
    "from langchain.chains.conversational_retrieval.prompts import CONDENSE_QUESTION_PROMPT\n",
    "from langchain.chains.question_answering import load_qa_chain\n",
    "from langchain.chains.qa_with_sources import load_qa_with_sources_chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "06cd8107-9097-4de1-b3d8-db11609d25bf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start at    ... 2024-03-12 01:39:12.341629\n",
      "\n",
      "Complete at ... 2024-03-12 01:39:39.696789\n",
      "Total time used 27.354386806488037 seconds\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from langchain.memory import ConversationBufferMemory\n",
    "# We can now create a memory object, which is neccessary to track the inputs/outputs and hold a conversation.\n",
    "memory = ConversationBufferMemory(memory_key=\"chat_history\", return_messages=True)\n",
    "\n",
    "bot = ConversationalRetrievalChain.from_llm(\n",
    "    llm, retriever, memory=memory, verbose=False\n",
    ")\n",
    "\n",
    "time1 = utility.chain_starts()\n",
    "\n",
    "result = bot.invoke({\"question\": query})\n",
    "\n",
    "utility.chain_completes(time1)\n",
    "\n",
    "# print(result,flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "afba9fa6-da38-4ae7-8889-3ea596b3c8c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Does company XYZ use structurizr dsl\n"
     ]
    }
   ],
   "source": [
    "print(result[\"question\"], flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1996ef97-6569-45f5-b4d3-3f380f8a2f41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Yes, according to the context provided, company XYZ uses Structurizr DSL for software architecture modeling and documentation.\n"
     ]
    }
   ],
   "source": [
    "print(result[\"answer\"], flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "711755be-289a-4386-a5d8-0e50c1001062",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start at    ... 2024-03-12 01:39:39.722983\n",
      "\n",
      "Complete at ... 2024-03-12 01:40:12.446788\n",
      "Total time used 32.72275114059448 seconds\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## Pass in chat history\n",
    "bot = ConversationalRetrievalChain.from_llm(\n",
    "    llm, retriever\n",
    ")\n",
    "\n",
    "time1 = utility.chain_starts()\n",
    "\n",
    "chat_history = []\n",
    "result = bot.invoke({\"question\": query, \"chat_history\": chat_history})\n",
    "\n",
    "utility.chain_completes(time1)\n",
    "\n",
    "# print(result,flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "812b2278-93c9-4062-81fa-a126683ed662",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Does company XYZ use structurizr dsl\n"
     ]
    }
   ],
   "source": [
    "print(result[\"question\"], flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "848ce8df-70ca-4032-8931-4efab84cfadd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Yes, according to the context provided, company XYZ uses Structurizr DSL for software architecture modeling and documentation.\n"
     ]
    }
   ],
   "source": [
    "print(result[\"answer\"], flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "d12669ff-6e48-45ed-8d57-72b60e6dd89b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start at    ... 2024-03-12 01:40:12.476953\n",
      "\n",
      "Complete at ... 2024-03-12 01:40:42.366396\n",
      "Total time used 29.888598203659058 seconds\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## Return Source Documents\n",
    "\n",
    "bot = ConversationalRetrievalChain.from_llm(\n",
    "    llm, retriever, return_source_documents=True\n",
    ")\n",
    "time1 = utility.chain_starts()\n",
    "\n",
    "chat_history = []\n",
    "result = bot.invoke({\"question\": query, \"chat_history\": chat_history})\n",
    "\n",
    "utility.chain_completes(time1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "0c916586-48b8-432e-b1f1-6dfbd0dd60de",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "answer:\n",
      " Yes, according to the context provided, company XYZ uses Structurizr DSL for software architecture modeling and documentation.\n"
     ]
    }
   ],
   "source": [
    "print(f'answer:\\n{result[\"answer\"]}',flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "c5589f93-ce04-47a0-b9d6-96f7d57cca7b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "source_documents:\n",
      "Some famous companies such as ABC and XYZ are using Structurizr DSL for software architecture modeling and documentation.\n"
     ]
    }
   ],
   "source": [
    "print(f'source_documents:\\n{result[\"source_documents\"][0].page_content}',flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "85d3057c-1135-46c9-a885-a0d0b8481893",
   "metadata": {},
   "outputs": [],
   "source": [
    "## ConversationalRetrievalChain with map_reduce\n",
    "\n",
    "# question_generator = LLMChain(llm=llm, prompt=CONDENSE_QUESTION_PROMPT)\n",
    "# doc_chain = load_qa_with_sources_chain(llm, chain_type=\"map_reduce\")\n",
    "\n",
    "# chain = ConversationalRetrievalChain(\n",
    "#     retriever=retriever,\n",
    "#     question_generator=question_generator,\n",
    "#     combine_docs_chain=doc_chain,\n",
    "# )\n",
    "\n",
    "# chat_history = []\n",
    "# result = chain.invoke({\"question\": query, \"chat_history\": chat_history})\n",
    "# print(result, flush=True)\n",
    "\n",
    "## ValueError: Document prompt requires documents to have metadata variables: ['source']. Received document with missing metadata: ['source']."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "84dbbaca-a40e-4376-b6cd-8a66b13bc707",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Revise PromptTemplate\n",
    "# from langchain_core.prompts import PromptTemplate\n",
    "\n",
    "# template = \"\"\"Use the following pieces of context to answer the question at the end. \n",
    "# If you don't know the answer, just say that you don't know, don't try to make up an answer. \n",
    "# Use three sentences maximum and keep the answer as concise as possible. \n",
    "# Always say \"thanks for asking!\" at the end of the answer. \n",
    "# {context}\n",
    "# Question: {question}\n",
    "# Helpful Answer:\"\"\"\n",
    "# prompt = PromptTemplate.from_template(template)\n",
    "\n",
    "# chain = (\n",
    "#     {\"context\": retriever, \"question\": RunnablePassthrough()}\n",
    "#     | prompt\n",
    "#     | llm\n",
    "# )\n",
    "\n",
    "# time1 = utility.chain_starts()\n",
    "\n",
    "# result = chain.invoke(query)\n",
    "\n",
    "# print(result, flush=True)\n",
    "\n",
    "# time1 = utility.chain_starts()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdde19af-7ffa-41d2-9645-b11c2a2b4ef5",
   "metadata": {},
   "source": [
    "## load_qa_with_sources_chain\n",
    "\n",
    "https://api.python.langchain.com/en/latest/chains/langchain.chains.qa_with_sources.loading.load_qa_with_sources_chain.html#langchain.chains.qa_with_sources.loading.load_qa_with_sources_chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "cee62dc5-3834-4433-ab65-39879eb48880",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Just for testing purpose to hard code \"source\"=\"abc\" to avoid the error below\n",
    "# ValueError: Document prompt requires documents to have metadata variables: ['source']. Received document with missing metadata: ['source'].\n",
    "docs = retriever.get_relevant_documents(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "f3cc9940-71e8-4698-83df-aa0ca6d5e78a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#print(f'{docs[0]}',flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "441340ea-962d-4adf-944f-82601cafecdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "for doc in docs:\n",
    "    doc.metadata[\"source\"]=\"abc\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87c96838-bb4a-4d01-a896-1b72b6d4c606",
   "metadata": {},
   "source": [
    "### run the query first time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "75c82b91-e0a6-4769-a8bf-557b3957f923",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start at    ... 2024-03-12 02:01:42.715093\n",
      "\n",
      "Complete at ... 2024-03-12 02:06:52.141264\n",
      "Total time used 309.42338728904724 seconds\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from langchain.chains.qa_with_sources import load_qa_with_sources_chain\n",
    "\n",
    "chain = load_qa_with_sources_chain(llm, chain_type=\"map_reduce\", return_intermediate_steps=True)\n",
    "\n",
    "time1 = utility.chain_starts()\n",
    "\n",
    "result = chain.invoke({\"input_documents\": docs, \"question\": query}, return_only_outputs=True)\n",
    "\n",
    "utility.chain_completes(time1)\n",
    "\n",
    "# print(result, flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "1ed52b6a-fb66-4481-bd76-06924aa28036",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[' \"Some famous companies such as ABC and XYZ are using Structurizz DSL for software architecture modeling and documentation.\"\\n\\nThis text confirms that Company XYZ is one of the companies mentioned as using Structurizr DSL. The text does not provide additional details about how or to what extent they are using it, but it does establish that they do use it for software architecture modeling and documentation.', ' \"Please see the DSL cookbook for a tutorial guide to the Structurizr DSL.\"\\n\\nThis text suggests that someone has referred to the Structurizr DSL (Domain Specific Language) and provided a resource for learning more about it. However, it does not directly answer whether company XYZ uses the Structurizr DSL or not.']\n"
     ]
    }
   ],
   "source": [
    "print(result[\"intermediate_steps\"], flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "3d03c2f4-f019-4adf-9684-c18f73e7c8e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Yes, according to the given sources, Company XYZ uses Structurizr DSL for software architecture modeling and documentation.\n",
      "SOURCES: abc\n"
     ]
    }
   ],
   "source": [
    "print(result[\"output_text\"], flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ed7c59f-ad57-4db2-8429-fbb3bf3354b5",
   "metadata": {},
   "source": [
    "### run the query again to see whether the Answer is consistent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "c1d2f76e-4bb8-4ecc-b31c-e487685a2671",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start at    ... 2024-03-12 02:06:52.168352\n",
      "\n",
      "Complete at ... 2024-03-12 02:11:44.938654\n",
      "Total time used 292.76894187927246 seconds\n",
      "\n"
     ]
    }
   ],
   "source": [
    "time1 = utility.chain_starts()\n",
    "\n",
    "result = chain.invoke({\"input_documents\": docs, \"question\": query}, return_only_outputs=True)\n",
    "\n",
    "utility.chain_completes(time1)\n",
    "\n",
    "# print(result, flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "a9a8d5e7-bba1-4275-b603-92623f2d7ef5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[' \"Some famous companies such as ABC and XYZ are using Structurizz DSL for software architecture modeling and documentation.\"\\n\\nTherefore, the answer to your question is: Yes, according to the provided text, company XYZ uses Structurizr DSL.', ' \"Please see the DSL cookbook for a tutorial guide to the Structurizr DSL.\"\\n\\nThis text suggests that someone has referred to the Structurizr DSL in this document. However, it does not directly answer the question of whether company XYZ uses Structurizr DSL specifically. To definitively answer the question, additional context or information would be required.']\n"
     ]
    }
   ],
   "source": [
    "print(result[\"intermediate_steps\"], flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "98a69a6d-7cd8-46a9-915a-8d5e44d0dace",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Yes, according to Source \"abc\", company XYZ is mentioned among the famous companies using Structurizr DSL for software architecture modeling and documentation.\n",
      "\n",
      "SOURCES: abc\n"
     ]
    }
   ],
   "source": [
    "print(result[\"output_text\"], flush=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49c5dd85-9b8d-4e3f-acaa-66b2bb2efe8f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
